Motor System-Dependent Effects of Amygdala and Ventral Striatum Lesions on Explore-Exploit Behaviors.

Deciding whether to forego immediate rewards or explore new opportunities is a key component of flexible behavior and is critical for the survival of the species. Although previous studies have shown that different cortical and subcortical areas, including the amygdala and ventral striatum (VS), are implicated in representing the immediate (exploitative) and future (explorative) value of choices, the effect of the motor system used to make choices has not been examined. Here, we tested male rhesus macaques with amygdala or VS lesions on two versions of a three-arm bandit task where choices were registered with either a saccade or an arm movement. In both tasks we presented the monkeys with explore-exploit tradeoffs by periodically replacing familiar options with novel options that had unknown reward probabilities. We found that monkeys explored more with saccades but showed better learning with arm movements. VS lesions caused the monkeys to be more explorative with arm movements and less explorative with saccades, although this may have been due to an overall decrease in performance. VS lesions affected the monkeys' ability to learn novel stimulus-reward associations in both tasks, while after amygdala lesions this effect was stronger when choices were made with saccades. Further, on average, VS and amygdala lesions reduced the monkeys' ability to choose better options only when choices were made with a saccade. These results show that learning reward value associations to manage explore-exploit behaviors is motor system dependent and they further define the contributions of amygdala and VS to reinforcement learning.

The amygdala is not necessary for the familiarity aspect of recognition memory.

Dual-process accounts of item recognition posit two memory processes: slow but detailed recollection, and quick but vague familiarity. It has been proposed, based on prior rodent work, that the amygdala is critical for the familiarity aspect of item recognition. Here, we evaluated this proposal in male rhesus monkeys (Macaca mulatta) with selective bilateral excitotoxic amygdala damage. We used four established visual memory tests designed to assess different aspects of familiarity, all administered on touchscreen computers. Specifically, we assessed monkeys' tendencies to make low-latency false alarms, to make false alarms to recently seen lures, to produce curvilinear ROC curves, and to discriminate stimuli based on repetition across days. Three of the four tests showed no familiarity impairment and the fourth was explained by a deficit in reward processing. Consistent with this, amygdala damage did produce an anticipated deficit in reward processing in a three-arm-bandit gambling task, verifying the effectiveness of the lesions. Together, these results contradict prior rodent work and suggest that the amygdala is not critical for the familiarity aspect of item recognition.

Efficient viral expression of a chemogenetic receptor in the old-world monkey amygdala.

Genetically encoded synthetic receptors, such as the chemogenetic and optogenetic proteins, are powerful tools for functional brain studies in animals. In the primate brain, with its comparatively large, intricate anatomical structures, it can be challenging to express transgenes, such as the hM4Di chemogenetic receptor, in a defined anatomical structure with high penetrance. Here, we compare parameters for lentivirus vector injections in the rhesus monkey amygdala. We find that four injections of 20 µl, infused at 0.5 µl/min, can achieve neuronal hM4Di expression in 50-100% of neurons within a 60 mm3 volume, without observable damage from overexpression. Increasing the number of hM4Di_CFP lentivirus injections to up to 12 sites per hemisphere, resulted in 30%-40% neuronal coverage of the overall amygdala volume, with coverage reaching 60% in some subnuclei. Manganese Chloride was mixed with lentivirus and used as an MRI marker to verify targeting accuracy and correct unsuccessful injections in these experiments. In a separate monkey we visualized, in vivo, viral expression of the hM4Di receptor protein in the amygdala, using Positron Emission Tomography. Together, these data show efficient and verifiable expression of a chemogenetic receptor in old-world monkey amygdala.

Adverse childhood experiences and hormonal contraception: Interactive impact on sexual reward function.

The current literature suggests that some women are uniquely vulnerable to negative effects of hormonal contraception (HC) on affective processes. However, little data exists as to which factors contribute to such vulnerability. The present study evaluated the impact of prepubertal adverse childhood experiences (ACEs) on reward processing in women taking HC (N = 541) compared to naturally cycling women (N = 488). Participants completed an online survey assessing current and past HC use and exposure to 10 different adverse childhood experiences (ACEs) before puberty (ACE Questionnaire), with participants categorized into groups of low (0-1) versus high (≥2) prepubertal ACE exposure. Participants then completed a reward task rating their expected and experienced valence for images that were either erotic, pleasant (non-erotic), or neutral. Significant interactions emerged between prepubertal ACE exposure and HC use on expected (p = 0.028) and experienced (p = 0.025) valence ratings of erotic images but not pleasant or neutral images. Importantly, follow-up analyses considering whether women experienced HC-induced decreases in sexual desire informed the significant interaction for expected valence ratings of erotic images. For current HC users, prepubertal ACEs interacted with HC-induced decreased sexual desire (p = 0.008), such that high ACE women reporting decreased sexual desire on HC showed substantially decreased ratings for anticipated erotic images compared to both high prepubertal ACE women without decreased sexual desire (p < 0.001) and low prepubertal ACE women also reporting decreased sexual desire (p = 0.010). The interaction was not significant in naturally cycling women reporting previous HC use, suggesting that current HC use could be impacting anticipatory reward processing of sexual stimuli among certain women (e.g., high prepubertal ACE women reporting HC-induced decreases in sexual desire). The study provides rationale for future randomized, controlled trials to account for prepubertal ACE exposure to promote contraceptive selection informed by behavioral evidence.

Electrophysiological Markers of Aberrant Cue-Specific Exploration in Hazardous Drinkers.

Background: Hazardous drinking is associated with maladaptive alcohol-related decision-making. Existing studies have often focused on how participants learn to exploit familiar cues based on prior reinforcement, but little is known about the mechanisms that drive hazardous drinkers to explore novel alcohol cues when their value is not known. Methods: We investigated exploration of novel alcohol and non-alcohol cues in hazardous drinkers (N = 27) and control participants (N = 26) during electroencephalography (EEG). A normative computational model with two free parameters was fit to estimate participants' weighting of the future value of exploration and immediate value of exploitation. Results: Hazardous drinkers demonstrated increased exploration of novel alcohol cues, and conversely, increased probability of exploiting familiar alternatives instead of exploring novel non-alcohol cues. The motivation to explore novel alcohol stimuli in hazardous drinkers was driven by an elevated relative future valuation of uncertain alcohol cues. P3a predicted more exploratory decision policies driven by an enhanced relative future valuation of novel alcohol cues. P3b did not predict choice behavior, but computational parameter estimates suggested that hazardous drinkers with enhanced P3b to alcohol cues were likely to learn to exploit their immediate expected value. Conclusions: Hazardous drinkers did not display atypical choice behavior, different P3a/P3b amplitudes, or computational estimates to novel non-alcohol cues-diverging from previous studies in addiction showing atypical generalized explore-exploit decisions with non-drug-related cues. These findings reveal that cue-specific neural computations may drive aberrant alcohol-related decision-making in hazardous drinkers-highlighting the importance of drug-relevant cues in studies of decision-making in addiction.

Recurrent activity within microcircuits of macaque dorsolateral prefrontal cortex tracks cognitive flexibility.

Human and non-human primate data clearly implicate the dorsolateral prefrontal cortex (dlPFC) as critical for advanced cognitive functions 1,2 . It is thought that intracortical synaptic architectures within dlPFC are the integral neurobiological substrate that gives rise to these processes, including working memory, inferential reasoning, and decision-making 3-7 . In the prevailing model, each cortical column makes up one fundamental processing unit composed of dense intrinsic connectivity, conceptualized as the 'canonical' cortical microcircuit 3,8 . Each cortical microcircuit receives sensory and cognitive information from a variety of sources which are represented by sustained activity within the microcircuit, referred to as persistent or recurrent activity 4,9 . Via recurrent connections within the microcircuit, activity can propagate for a variable length of time, thereby allowing temporary storage and computations to occur locally before ultimately passing a transformed representation to a downstream output 4,5,10 . Competing theories regarding how microcircuit activity is coordinated have proven difficult to reconcile in vivo where intercortical and intracortical computations cannot be fully dissociated 5,9,11,12 . Here, we interrogated the intrinsic features of isolated microcircuit networks using high-density calcium imaging of macaque dlPFC ex vivo . We found that spontaneous activity is intrinsically maintained by microcircuit architecture, persisting at a high rate in the absence of extrinsic connections. Further, using perisulcal stimulation to evoke persistent activity in deep layers, we found that activity propagates through stochastically assembled intracortical networks, creating predictable population-level events from largely non-overlapping ensembles. Microcircuit excitability covaried with individual cognitive performance, thus anchoring heuristic models of abstract cortical functions within quantifiable constraints imposed by the underlying synaptic architecture.

What Does the Frontopolar Cortex Contribute to Goal-Directed Cognition and Action?

Understanding the unique functions of different subregions of primate prefrontal cortex has been a longstanding goal in cognitive neuroscience. Yet, the anatomy and function of one of its largest subregions (the frontopolar cortex) remain enigmatic and underspecified. Our Society for Neuroscience minisymposium Primate Frontopolar Cortex: From Circuits to Complex Behaviors will comprise a range of new anatomic and functional approaches that have helped to clarify the basic circuit anatomy of the frontal pole, its functional involvement during performance of cognitively demanding behavioral paradigms in monkeys and humans, and its clinical potential as a target for noninvasive brain stimulation in patients with brain disorders. This review consolidates knowledge about the anatomy and connectivity of frontopolar cortex and provides an integrative summary of its function in primates. We aim to answer the question: what, if anything, does frontopolar cortex contribute to goal-directed cognition and action?

The neurocomputational bases of explore-exploit decision-making.

Flexible decision-making requires animals to forego immediate rewards (exploitation) and try novel choice options (exploration) to discover if they are preferable to familiar alternatives. Using the same task and a partially observable Markov decision process (POMDP) model to quantify the value of choices, we first determined that the computational basis for managing explore-exploit tradeoffs is conserved across monkeys and humans. We then used fMRI to identify where in the human brain the immediate value of exploitative choices and relative uncertainty about the value of exploratory choices were encoded. Consistent with prior neurophysiological evidence in monkeys, we observed divergent encoding of reward value and uncertainty in prefrontal and parietal regions, including frontopolar cortex, and parallel encoding of these computations in motivational regions including the amygdala, ventral striatum, and orbitofrontal cortex. These results clarify the interplay between prefrontal and motivational circuits that supports adaptive explore-exploit decisions in humans and nonhuman primates.

Differential coding of goals and actions in ventral and dorsal corticostriatal circuits during goal-directed behavior.

Goal-directed behavior requires identifying objects in the environment that can satisfy internal needs and executing actions to obtain those objects. The current study examines ventral and dorsal corticostriatal circuits that support complementary aspects of goal-directed behavior. We analyze activity from the amygdala, ventral striatum, orbitofrontal cortex, and lateral prefrontal cortex (LPFC) while monkeys perform a three-armed bandit task. Information about chosen stimuli and their value is primarily encoded in the amygdala, ventral striatum, and orbitofrontal cortex, while the spatial information is primarily encoded in the LPFC. Before the options are presented, information about the to-be-chosen stimulus is represented in the amygdala, ventral striatum, and orbitofrontal cortex; at the time of choice, the information is passed to the LPFC to direct a saccade. Thus, learned value information specifying behavioral goals is maintained throughout the ventral corticostriatal circuit, and it is routed through the dorsal circuit at the time actions are selected.

A pragmatic reevaluation of the efficacy of nonhuman primate optogenetics for psychiatry.

Translational neuroscience is committed to generating discoveries in the laboratory that ultimately can improve human lives. Optogenetics has received considerable attention because of its demonstrated promise in rodent brains to manipulate cells and circuits. In a recent report, Tremblay et al. [28] introduce an open resource detailing optogenetic studies of the nonhuman primate (NHP) brain and make robust claims about the translatability of the technology. We propose that their quantitative (e.g. a 91% success rate) and theoretical claims are questionable because the data were analyzed at a level relevant to the rodent but not NHP brain. Injections were clustered within a few monkeys in a few studies in a few brain regions, and their definitions of success were not clearly relevant to human neuropsychiatric disease. A reanalysis of the data with a modified definition of success that included a behavioral and biological effect revealed a 62.5% success rate that was lower when considering only strong outcomes (53.1%). This calls into question the current efficacy of optogenetic techniques in the NHP brain and suggests that we are a long way from being able to leverage them in 'the service of patients with neurological or psychiatric conditions' as the Tremblay report claims.

Entropy-based metrics for predicting choice behavior based on local response to reward.

For decades, behavioral scientists have used the matching law to quantify how animals distribute their choices between multiple options in response to reinforcement they receive. More recently, many reinforcement learning (RL) models have been developed to explain choice by integrating reward feedback over time. Despite reasonable success of RL models in capturing choice on a trial-by-trial basis, these models cannot capture variability in matching behavior. To address this, we developed metrics based on information theory and applied them to choice data from dynamic learning tasks in mice and monkeys. We found that a single entropy-based metric can explain 50% and 41% of variance in matching in mice and monkeys, respectively. We then used limitations of existing RL models in capturing entropy-based metrics to construct more accurate models of choice. Together, our entropy-based metrics provide a model-free tool to predict adaptive choice behavior and reveal underlying neural mechanisms.

Adolescent Dopamine Neurons Represent Reward Differently during Action and State Guided Learning.

Neuronal underpinning of learning cause-and-effect associations in the adolescent brain remains poorly understood. Two fundamental forms of associative learning are Pavlovian (classical) conditioning, where a stimulus is followed by an outcome, and operant (instrumental) conditioning, where outcome is contingent on action execution. Both forms of learning, when associated with a rewarding outcome, rely on midbrain dopamine neurons in the ventral tegmental area (VTA) and substantia nigra (SN). We find that, in adolescent male rats, reward-guided associative learning is encoded differently by midbrain dopamine neurons in each conditioning paradigm. Whereas simultaneously recorded VTA and SN adult neurons have a similar phasic response to reward delivery during both forms of conditioning, adolescent neurons display a muted reward response during operant but a profoundly larger reward response during Pavlovian conditioning. These results suggest that adolescent neurons assign a different value to reward when it is not gated by action. The learning rate of adolescents and adults during both forms of conditioning was similar, supporting the notion that differences in reward response in each paradigm may be because of differences in motivation and independent of state versus action value learning. Static characteristics of dopamine neurons, such as dopamine cell number and size, were similar in the VTA and SN of both ages, but there were age-related differences in stimulated dopamine release and correlated spike activity, suggesting that differences in reward responsiveness by adolescent dopamine neurons are not because of differences in intrinsic properties of these neurons but engagement of different dopaminergic networks.SIGNIFICANCE STATEMENT Reckless behavior and impulsive decision-making by adolescents suggest that motivated behavioral states are encoded differently by the adolescent brain. Motivated behavior, which is dependent on the function of the dopamine system, follows learning of cause-and-effect associations in the environment. We find that dopamine neurons in adolescents encode reward differently depending on the cause-and-effect relationship of the means to receive that reward. Compared with adults, reward contingent on action led to a muted response, whereas reward that followed a cue but was not gated by action produced an augmented phasic response. These data demonstrate an age-related difference in dopamine neuron response to reward that is not uniform and is guided by processes that differentiate between state and action values.

Anterior cingulate and putamen neurons flexibly learn whether a hot dog is a sandwich.

Policy search lets you discover rules and adapt behavior. In this issue of Neuron, Cohen et al. (2021) demonstrate that the dynamics of neurons in primate anterior cingulate cortex and putamen indicate when a correct policy is discovered and confidence in executing decisions under that policy.

Effects of Amygdala Lesions on Object-Based Versus Action-Based Learning in Macaques.

The neural systems that underlie reinforcement learning (RL) allow animals to adapt to changes in their environment. In the present study, we examined the hypothesis that the amygdala would have a preferential role in learning the values of visual objects. We compared a group of monkeys (Macaca mulatta) with amygdala lesions to a group of unoperated controls on a two-armed bandit reversal learning task. The task had two conditions. In the What condition, the animals had to learn to select a visual object, independent of its location. And in the Where condition, the animals had to learn to saccade to a location, independent of the object at the location. In both conditions choice-outcome mappings reversed in the middle of the block. We found that monkeys with amygdala lesions had learning deficits in both conditions. Monkeys with amygdala lesions did not have deficits in learning to reverse choice-outcome mappings. Rather, amygdala lesions caused the monkeys to become overly sensitive to negative feedback which impaired their ability to consistently select the more highly valued action or object. These results imply that the amygdala is generally necessary for RL.

Balancing exploration and exploitation with information and randomization.

Explore-exploit decisions require us to trade off the benefits of exploring unknown options to learn more about them, with exploiting known options, for immediate reward. Such decisions are ubiquitous in nature, but from a computational perspective, they are notoriously hard. There is therefore much interest in how humans and animals make these decisions and recently there has been an explosion of research in this area. Here we provide a biased and incomplete snapshot of this field focusing on the major finding that many organisms use two distinct strategies to solve the explore-exploit dilemma: a bias for information ('directed exploration') and the randomization of choice ('random exploration'). We review evidence for the existence of these strategies, their computational properties, their neural implementations, as well as how directed and random exploration vary over the lifespan. We conclude by highlighting open questions in this field that are ripe to both explore and exploit.

Assessing the role of the amygdala in fear of pain: Neural activation under threat of shock.

INTRODUCTION: The DSM-5 explicitly states that the neural system model of specific phobia is centered on the amygdala. However, this hypothesis is predominantly supported by human studies on animal phobia, whereas visual cuing of other specific phobias, such as dental fear, do not consistently show amygdala activation. Considering that fear of anticipated pain is one of the best predictors of dental phobia, the current study investigated neural and autonomic activity of pain anticipation in individuals varying in the degree of fear of dental pain. METHOD: Functional brain activity (fMRI) was measured in women (n = 31) selected to vary in the degree of self-reported fear of dental pain when under the threat of shock, in which one color signaled the possibility of receiving a painful electric shock and another color signaled safety. RESULTS: Enhanced functional activity during threat, compared to safety, was found in regions including anterior insula and anterior/mid cingulate cortex. Importantly, threat reactivity in the anterior insula increased as reported fear of pain increased and further predicted skin conductance changes during pain anticipation. LIMITATIONS: The sample was comprised of women. CONCLUSIONS: Individual differences in fear of pain vary with activation in the anterior insula, rather than with the amygdala, indicating that fear is not uniquely associated with amygdala activation. Whereas coping techniques such as emotion regulation have been found to vary with activation in a frontal-amygdala circuit when confronted with visual cues, precision psychiatry may need to target specific brain circuits to diagnose and treat different types of specific phobia.

Aversive perception in a threat context: Separate and independent neural activation.

Unpleasant, compared to neutral, scenes reliably prompt enhanced functional brain activity in the amygdala and inferotemporal cortex. Considering data from psychophysiological studies in which defensive reactivity is further enhanced when viewing unpleasant scenes under threat of shock (compared to safety), the current study investigates functional activation in the amygdala-inferotemporal circuit when unpleasant (or neutral) scenes are viewed under threat of shock or safety. In this paradigm, a cue signaling threat or safety was presented in conjunction with either an unpleasant or neutral picture. Replicating previous studies, unpleasant, compared to neutral, scenes reliably enhanced activation in the amygdala and inferotemporal cortex. Functional activity in these regions, however, did not differ whether scenes were presented in a context threatening shock exposure, compared to safety, which instead activated regions of the anterior insula and cingulate cortex. Taken together, the data support a view in which neural regions activated in different defensive situations act independently.